EMOTION RECOGNITION

FERNet: Fine-grained Extraction and Reasoning Network for Emotion Recognition in Dialogues (2020.aacl-main)

Challenge:	a human would recognize the emotion of an interlocutor and respond with an appropriate emotion, such as empathy and comfort.
Approach:	They propose to build a dialogue corpus annotated with two kinds of emotions . they collect tweets and annotate them with the emotion they put into the utterance .
Outcome:	The proposed method shows that it is difficult to recognize experienced emotions and multitask learning is effective.

Copied to clipboard

Yingmei Guo, Zhiyong Wu, Mingxing Xu

Reasoning

Challenge:	Existing methods for emotion recognition in dialogues do not consider the content of the target utterance.
Approach:	They propose to model historical utterances without considering the content of the target utterant . they propose to use a fine-grained reasoning network to generate target-specific historical .
Outcome:	The proposed method achieves competitive performance compared with previous methods.

Multimodal Language Analysis with Recurrent Multistage Fusion (D18-1)

Copied to clipboard

Paul Pu Liang, Ziyin Liu, AmirAli Bagher Zadeh, Louis-Philippe Morency

DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversation (D19-1)

Challenge:	Comprehending multimodal language requires modeling interactions between modalities and between them.
Approach:	They propose a multistage fusion network which decomposes the fusion problem into multiple stages, each focused on a subset of multimodal signals for specialized, effective fusion.
Outcome:	The proposed model performs state-of-the-art across three datasets relating to multimodal sentiment analysis, emotion recognition, and speaker traits recognition.

Copied to clipboard

Deepanway Ghosal, Navonil Majumder, Soujanya Poria, Niyati Chhaya, Alexander Gelbukh

ADVISER: A Toolkit for Developing Multi-modal, Multi-domain and Socially-engaged Conversational Agents (2020.acl-demos)

Challenge:	Emotion recognition in conversation (ERC) has received much attention lately due to its potential widespread applications in diverse areas, such as health-care, education, and human resources.
Approach:	They propose a graph neural network-based approach to emotion recognition in conversation that leverages self and inter-speaker dependency of the interlocutors to model conversational context.
Outcome:	The proposed method outperforms the current state-of-the-art on a number of benchmark emotion classification datasets while minimizing context propagation issues.

Copied to clipboard

Chia-Yu Li, Daniel Ortega, Dirk Väth, Florian Lux, Lindsey Vanderlyn, Maximilian Schmidt, Michael Neumann, Moritz Völkel, Pavel Denisov, Sabrina Jenne, Zorica Kacarevic, Ngoc Thang Vu

Static and Dynamic Speaker Modeling based on Graph Neural Network for Emotion Recognition in Conversation (2022.naacl-srw)

Challenge:	Existing toolkits for developing dialog systems are limited to core components and do not support multi-modal processing and social signals.
Approach:	They propose to use ADVISER to develop multi-modal dialog agents using multi-text and social signals.
Outcome:	The proposed toolkit is flexible, easy to use, and easy to extend for linguists and cognitive scientists, thereby providing a flexible platform for collaborative research.

Copied to clipboard

Prakhar Saxena, Yin Jou Huang, Sadao Kurohashi

Graph Based Network with Contextualized Representations of Turns in Dialogue (2021.emnlp-main)

Challenge:	Hence, speaker modeling is important for the task of emotion recognition in conversation (ERC).
Approach:	They propose a graph-based ERC model which considers conversational context and speaker personality.
Outcome:	The proposed model outperforms baseline and other graph-based methods on a benchmark dataset.

Copied to clipboard

Bongseok Lee, Yong Suk Choi

Structured Data in NLP

Challenge:	Dialogue-based relation extraction (RE) aims to extract relation(s) between two arguments that appear in a dialogue.
Approach:	They propose a dialogue-based relation extraction model which is based on emotion recognition in conversations.
Outcome:	The proposed model outperforms the state-of-the-art models on most of the benchmark datasets.

MMM: An Emotion and Novelty-aware Approach for Multilingual Multimodal Misinformation Detection (2022.findings-aacl)

Copied to clipboard

Vipin Gupta, Rina Kumari, Nischal Ashok, Tirthankar Ghosal, Asif Ekbal

Reasoning

Challenge:	Increasing presence of multimedia content on the web promotes misinformation . detecting this category of misleading information is almost impossible without prior knowledge .
Approach:	They propose a novel multilingual multimodal misinformation dataset that includes background knowledge of misleading articles.
Outcome:	The proposed model outperforms the state-of-the-art on misinformation detection task.

MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations (P19-1)

Copied to clipboard

Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, Rada Mihalcea

Modelling Variability in Human Annotator Simulation (2024.findings-acl)

Challenge:	Emotion recognition in conversations has gained popularity due to its potential applications. Until now, a large multimodal multi-party emotional conversational database containing more than two speakers per dialogue was missing.
Approach:	They propose to extend and enhance EmotionLines by combining 13,000 utterances from Friends dialogues with emotion and sentiment labels.
Outcome:	The proposed dataset contains about 13,000 utterances from 1,433 dialogues from the TV-series Friends.

Copied to clipboard

Wen Wu, Wenlin Chen, Chao Zhang, Phil Woodland

Green & Sustainable NLP

Challenge:	Human annotator simulation (HAS) is a cost-effective alternative to human evaluation tasks.
Approach:	They propose a framework to model human annotation variability via meta-learning . conditional softmax flow model leverages diverse human annotations via meta learning . results demonstrate that method can predict aggregated behaviours of human annotators .
Outcome:	The proposed method achieves state-of-the-art performance on two real-world human evaluation tasks: emotion recognition and toxic speech detection.

Improving Empathetic Response Generation by Recognizing Emotion Cause in Conversations (2021.findings-emnlp)

Copied to clipboard

Jun Gao, Yuhan Liu, Haolin Deng, Wei Wang, Yu Cao, Jiachen Du, Ruifeng Xu

Multimodal fusion via cortical network inspired losses (2022.acl-long)

Challenge:	Existing approaches to empathetic response generation ignore the emotion cause . existing dialogue systems lack emotion understanding and empathy .
Approach:	They propose a framework that integrates emotion cause information into empathetic response generation by predicting context emotion labels and sequence of emotion cause-oriented labels.
Outcome:	The proposed framework improves empathetic response generation by incorporating emotion cause information into the model.

Copied to clipboard

Shiv Shankar

Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition (2024.acl-long)

Challenge:	Recent work in deep fusion models has led to substantial improvements over unimodal approaches in areas like speech recognition, emotion recognition and analysis.
Approach:	They propose to introduce neural dependencies into the loss functions to allow for fusion of different modalities while keeping the model complexity manageable.
Outcome:	Experiments on multimodal sentiment analysis tasks show that the proposed approach provides a consistent performance boost.

Copied to clipboard

Zirun Guo, Tao Jin, Zhou Zhao

Multimodal and Multi-view Models for Emotion Recognition (P19-1)

Challenge:	Existing methods for multimodal sentiment analysis often fail due to equipment failure, data corruption, privacy issues and the like.
Approach:	They propose a multimodal Transformer framework using prompt learning to address the issue of missing modalities.
Outcome:	The proposed framework outperforms existing methods significantly across evaluation metrics.

Copied to clipboard

Gustavo Aguilar, Viktor Rozgic, Weiran Wang, Chao Wang

Evaluating Vision-Language Models for Emotion Recognition (2025.findings-naacl)

Challenge:	combining lexical and acoustic information results in more robust and accurate models . combining both modalities may be a bottleneck in a deployment pipeline due to computational complexity or privacy constraints .
Approach:	They propose to combine acoustic and lexical information to provide a deployable acustic model . they use multimodal models and two attention mechanisms to assess the benefits of lexicals .
Outcome:	The proposed model outperforms the state-of-the-art on the USC-IEMOCAP dataset . it significantly surpasses models that have been exclusively trained with acoustic features .

Copied to clipboard

Sree Bhattacharyya, James Z. Wang

Directed Acyclic Graph Network for Conversational Emotion Recognition (2021.acl-long)

Challenge:	Large Vision-Language Models (VLMs) have been used for objective multimodal reasoning tasks for decades.
Approach:	They present a comprehensive evaluation of large vision-language models for recognizing evoked emotions from images.
Outcome:	The proposed model performs well in evoked emotion recognition task and is robust to human errors.

Copied to clipboard

Weizhou Shen, Siyue Wu, Yunyi Yang, Xiaojun Quan

EmotionQueen: A Benchmark for Evaluating Empathy of Large Language Models (2024.findings-acl)

Challenge:	Empirical evidence shows that a good representation of conversation context significantly contributes to the model performance.
Approach:	They propose to encode query utterances with a directed acyclic graph to better model the intrinsic structure within a conversation.
Outcome:	The proposed model outperforms existing models on four ERC benchmarks with state-of-the-art models employed as baselines.

Copied to clipboard

Yuyan Chen, Songzhou Yan, Sijia Liu, Yueze Li, Yanghua Xiao

The Correlation Between Emotion in Text and Speech Segments is Limited: A Cross-Modal Study (2026.findings-eacl)

Challenge:	Existing evaluations of emotional intelligence in large language models (LLMs) focus on basic sentiment analysis tasks, such as emotion recognition, which is not enough to evaluate LLMs’ overall emotional intelligence.
Approach:	They propose a framework for evaluating the emotional intelligence of large language models (LLMs) that includes four distinct tasks: Key Event Recognition, Mixed Event Recognition and Implicit Emotional Recognition.
Outcome:	The proposed framework includes four distinct tasks: Key Event Recognition, Mixed Event Recognition and Implicit Emotional Recognition.

Copied to clipboard

David Lindevelt, Suzan Verberne, Joost Broekens

Seeing is Believing: Emotion-Aware Audio-Visual Language Modeling for Expressive Speech Generation (2025.findings-emnlp)

Challenge:	a recent study has shown that text-to-speech systems can capture human-like emotion, but they lack the ability to predict emotion in speech.
Approach:	They propose to use 8 large language models for identifying emotion in text and 2 audio models for emotion in speech to investigate the correlation between emotion and speech.
Outcome:	The proposed models perform well on emotion recognition from situational text and audiobooks, but show weak correlation for Valence only.

Copied to clipboard

Weiting Tan, Jiachen Lian, Hirofumi Inaguma, Paden Tomasello, Philipp Koehn, Xutai Ma

Multimodal Routing: Improving Local and Global Interpretability of Multimodal Language Analysis (2020.emnlp-main)

Challenge:	AVLM integrates full-face visual cues into a pre-trained expressive speech model.
Approach:	They propose an Audio-Visual Language Model (AVLM) for expressive speech generation by integrating full-face visual cues into a pre-trained expressive speech model.
Outcome:	The proposed model incorporates full-face visual cues into a pre-trained expressive speech model.

Copied to clipboard

Yao-Hung Hubert Tsai, Martin Ma, Muqiao Yang, Ruslan Salakhutdinov, Louis-Philippe Morency

Global-Local Modeling with Prompt-Based Knowledge Enhancement for Emotion Inference in Conversation (2023.findings-eacl)

Challenge:	Recent multimodal learning models with strong performances on human-centric tasks are often black-box with very limited interpretability.
Approach:	They propose a multimodal routing algorithm which dynamically adjusts weights between input and output modalities for each input sample.
Outcome:	The proposed model can interpret modality-prediction relationships globally and locally for each input sample while keeping competitive performance compared to state-of-the-art methods.

Copied to clipboard

Renxi Wang, Shi Feng

The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages (2023.emnlp-main)

Challenge:	Existing studies on emotion recognition focus on recognizing emotions through a speaker’s utterance, while research on emotion inference predicts emotions of addressees through previous utterations.
Approach:	They propose a global-local modeling method based on recurrent neural networks and pre-trained language models to do emotion inference in conversation.
Outcome:	The proposed method achieves state-of-the-art on three datasets.

Copied to clipboard

Chiyu Zhang, Khai Doan, Qisheng Liao, Muhammad Abdul-Mageed

IIIT-H TEMD Semi-Natural Emotional Speech Database from Professional Actors and Non-Actors (2020.lrec-1)

Challenge:	Existing instruction tuned large language models (LLMs) struggle to understand cross-lingual sociopragmatic meaning (SM) lack of comprehensive investigation into their ability to understand SM is partly due to SM not being adequately represented in any of the existing benchmarks.
Approach:	They evaluate the performance of instruction tuned large language models (LLMs) on a multilingual benchmark specifically designed for SM understanding.
Outcome:	The proposed benchmark outperforms instruction tuned large language models on a wide range of tasks but falls behind task-specific finetuned models.

Copied to clipboard

Banothu Rambabu, Kishore Kumar Botsa, Gangamohan Paidi, Suryakanth V Gangashetty

Contextual Dependencies in Time-Continuous Multidimensional Affect Recognition (L18-1)

Challenge:	Existing databases for emotion recognition are limited due to privacy and legal issues.
Approach:	They propose to collect emotional speech data from actors and non-actors using designed drama situations and annotate them manually using a hybrid strategy.
Outcome:	The proposed database is based on simulated parallel, semi-natural, and (near to) natural databases.

Copied to clipboard

Dmitrii Fedotov, Denis Ivanko, Maxim Sidorov, Wolfgang Minker

Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (2021.acl-long)

Challenge:	despite of the research done in this area there is still no agreement on this issue.
Approach:	a paper compares the amount of context used in a model and performance of a time-continuous labelled spontaneous interaction.
Outcome:	a new study shows that the amount of context used in a model and performance is similar across models . the results show that knowledge about an appropriate context can reduce complexity and flexibility .

Copied to clipboard

Jinming Zhao, Ruichen Li, Qin Jin

Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph (P18-1)

Challenge:	Existing multimodal fusion models trained on full-modality samples fail when partial modalities are missing.
Approach:	They propose a model to deal with the uncertain missing modality problem by learning robust joint multimodal representations that can predict the representation of any missing modal given available modalities under different missing-modality conditions.
Outcome:	The proposed model significantly improves performance under uncertain missing-modality testing conditions and full-modalities ideal testing conditions.

Copied to clipboard

AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, Louis-Philippe Morency

Efficient Low-rank Multimodal Fusion With Modality-Specific Factors (P18-1)

Challenge:	Analyzing human multimodal language is emerging area of research in NLP.
Approach:	They propose a multimodal fusion technique to exploit how modalities interact in multimodal language.
Outcome:	The proposed technique exploits how modalities interact with each other in human multimodal language.

Copied to clipboard

Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, AmirAli Bagher Zadeh, Louis-Philippe Morency

MUSER: MUltimodal Stress detection using Emotion Recognition as an Auxiliary Task (2021.naacl-main)

Challenge:	Multimodal research is a growing field of artificial intelligence, and fusion is one of the main research problems.
Approach:	They propose a low-rank multimodal fusion method which integrates multiple unimodal representations into one compact multimodal representation.
Outcome:	The proposed method achieves competitive results on multimodal sentiment analysis, speaker trait analysis, and emotion recognition tasks while reducing computational complexity.

Copied to clipboard

Yiqun Yao, Michalis Papakostas, Mihai Burzo, Mohamed Abouelenien, Rada Mihalcea

A Study on the Ambiguity in Human Annotation of German Oral History Interviews for Perceived Emotion Recognition and Sentiment Analysis (2022.lrec-1)

Challenge:	Existing methods to detect stress have not explored the inter-dependence between emotion and stress.
Approach:	They propose a transformer-based model architecture and a novel multi-task learning algorithm with speed-based dynamic sampling strategy to improve stress detection.
Outcome:	The proposed model is effective with internal and external auxiliary tasks and achieves state-of-the-art results.

Copied to clipboard

Michael Gref, Nike Matthiesen, Sreenivasa Hikkal Venugopala, Shalaka Satheesh, Aswinkumar Vijayananth, Duc Bach Ha, Sven Behnke, Joachim Köhler

COSMIC: COmmonSense knowledge for eMotion Identification in Conversations (2020.findings-emnlp)

Challenge:	Sentiment analysis and emotion recognition can help research in audiovisual interview archives . however, humans perceive sentiments and emotions ambiguously and subjectively .
Approach:	They investigate human perceptions of emotions and sentiments in oral history interviews . they show that human perception for different emotions is ambiguous and subjective . authors propose deep learning as a way to categorize and search emotions .
Outcome:	The proposed techniques can be used to search and index audiovisual interviews . the authors show that human perceptions differ for different emotions .

Copied to clipboard

Deepanway Ghosal, Navonil Majumder, Alexander Gelbukh, Rada Mihalcea, Soujanya Poria

Commonsense Reasoning

Challenge:	Current methods for emotion recognition in conversations often face difficulties in context propagation, emotion shift detection, and differentiating between related emotion classes.
Approach:	They propose a framework that incorporates mental states, events, and causal relations to learn interactions between interlocutors participating in a conversation.
Outcome:	The proposed framework improves on four conversational benchmark datasets.

Visual Prompting in LLMs for Enhancing Emotion Recognition (2024.emnlp-main)

Copied to clipboard

Qixuan Zhang, Zhifeng Wang, Dylan Zhang, Wenjia Niu, Sabrina Caldwell, Tom Gedeon, Yang Liu, Zhenyue Qin

Adaptive Fusion Techniques for Multimodal Data (2021.eacl-main)

Challenge:	Existing methods for enhancing in-context emotion classification fail to include spatial relationships between different people and facial features within a single face.
Approach:	They propose a set-of-vision prompting approach that uses spatial information to mark targets precisely.
Outcome:	The proposed approach improves face count and emotion categorization while preserving the enriched image context.

Copied to clipboard

Gaurav Sahu, Olga Vechtomova

Emotion-Anchored Contrastive Learning Framework for Emotion Recognition in Conversation (2024.findings-naacl)

Challenge:	Effective fusion of data from multiple modalities is challenging due to the heterogeneous nature of multimodal data.
Approach:	They propose two adaptive fusion techniques that aim to combine multimodal data effectively.
Outcome:	The proposed networks can model context from other modalities better than existing methods.

Copied to clipboard

Fangxu Yu, Junjie Guo, Zhen Wu, Xinyu Dai

UniMEEC: Towards Unified Multimodal Emotion Recognition and Emotion Cause (2024.findings-emnlp)

Challenge:	Emotion Recognition in Conversation (ERC) is a task that aims to identify the emotions behind each utterance in a conversation.
Approach:	They propose an Emotion-Anchored Contrastive Learning framework that generates more distinguishable utterance representations for similar emotions.
Outcome:	The proposed framework achieves state-of-the-art on similar emotions and performs well on similar ones.

Copied to clipboard

Guimin Hu, Zhihong Zhu, Daniel Hershcovich, Lijie Hu, Hasti Seifi, Jiayuan Xie

ViSoBERT: A Pre-Trained Language Model for Vietnamese Social Media Text Processing (2023.emnlp-main)

Challenge:	Existing studies treat emotion recognition and emotion cause extraction as two individual problems, ignoring their natural causality.
Approach:	They propose a Unified Multimodal Emotion recognition and Emotion-Cause analysis framework to explore the causality between emotion and emotion cause.
Outcome:	The proposed framework reformulates MERC and MECPE tasks as mask prediction problems and unifies them with a causal prompt template.

Copied to clipboard

Nam Nguyen, Thang Phan, Duc-Vu Nguyen, Kiet Nguyen

EmoBench: Evaluating the Emotional Intelligence of Large Language Models (2024.acl-long)

Challenge:	English and Chinese have seen the strong development of transformer-based language models for natural language processing tasks.
Approach:	They present a monolingual pre-trained language model for Vietnamese social media texts . they explore emotion recognition, hate speech detection, sentiment analysis, spam reviews detection .
Outcome:	The proposed model outperforms the existing models on Vietnamese social media tasks with fewer parameters.

Copied to clipboard

Sahand Sabour, Siyang Liu, Zheyuan Zhang, June Liu, Jinfeng Zhou, Alvionna Sunaryo, Tatia Lee, Rada Mihalcea, Minlie Huang

Supervised Prototypical Contrastive Learning for Emotion Recognition in Conversation (2022.emnlp-main)

Challenge:	Existing benchmarks for Emotional Intelligence (EI) focus on emotion recognition, neglecting essential EI capabilities.
Approach:	They propose a benchmark that proposes a comprehensive definition for machine EI . they propose 400 hand-crafted questions in English and Chinese to evaluate EI.
Outcome:	The proposed benchmarks focus on emotion recognition, neglecting EI capabilities . they are constructed from existing datasets, which include frequent patterns and errors . the proposed benchmark includes questions in English and Chinese that require thorough reasoning and understanding .

Copied to clipboard

Xiaohui Song, Longtao Huang, Hui Xue, Songlin Hu

Multi-Task Learning and Adapted Knowledge Models for Emotion-Cause Extraction (2021.findings-acl)

Challenge:	Existing methods to capture emotions in conversation (ERC) lack the correlation between emotions and semantics, resulting in many challenges.
Approach:	They propose a Supervised Prototypical Contrastive Learning (SPCL) loss for the ERC task . they use a Prototype Network to leverage the supervised contrastive learning approach .
Outcome:	The proposed approach outperforms CoG-BART's proposed approach on three widely used benchmarks and shows that it is effective on multiple scenarios.

Copied to clipboard

Elsbeth Turcan, Shuai Wang, Rishita Anubhai, Kasturi Bhattacharjee, Yaser Al-Onaizan, Smaranda Muresan

Low-Resource NLP

Challenge:	Detecting what emotions are expressed in text is a well-studied problem in natural language processing.
Approach:	They propose methods that combine common-sense knowledge with multi-task learning to perform joint emotion classification and emotion cause tagging.
Outcome:	The proposed models improve on both tasks when using common-sense reasoning and a multitask framework.

M3ED: Multi-modal Multi-scene Multi-label Emotional Dialogue Database (2022.acl-long)

Copied to clipboard

Jinming Zhao, Tenggan Zhang, Jingwen Hu, Yuchen Liu, Qin Jin, Xinchao Wang, Haizhou Li

Knowledge Aware Emotion Recognition in Textual Conversations via Multi-Task Incremental Transformer (2020.coling-main)

Challenge:	Existing data resources to support multimodal affective analysis in dialogues are limited in scale and diversity.
Approach:	They propose a multimodal multi-scene multi-label Emotional Dialogue dataset, M3ED, which contains 990 dyadic emotional dialogues from 56 different TV series.
Outcome:	The proposed dataset contains 990 dyadic emotional dialogues from 56 different TV series, a total of 9,082 turns and 24,449 utterances.

Copied to clipboard

Duzhen Zhang, Xiuyi Chen, Shuang Xu, Bo Xu

APPReddit: a Corpus of Reddit Posts Annotated for Appraisal (2022.lrec-1)

Challenge:	Existing models for ERTC use a few non-neutral categories to identify the emotion of each utterance.
Approach:	They propose a novel Knowledge Aware Incremental Transformer with Multi-task Learning to address these challenges by leveraging commonsense knowledge to leverage context.
Outcome:	The proposed model outperforms state-of-the-art models across five benchmark datasets.

Copied to clipboard

Marco Antonio Stranisci, Simona Frenda, Eleonora Ceccaldi, Valerio Basile, Rossana Damiano, Viviana Patti

CoMPM: Context Modeling with Speaker’s Pre-trained Memory Tracking for Emotion Recognition in Conversation (2022.naacl-main)

Challenge:	Existing resources for emotion recognition are lacking for appraisal models.
Approach:	They propose to use APPReddit to annotate non-experimental data according to Appraisal theories . they compare it with enISEAR, a corpus of events created in an experimental setting and annotated according to this theory.
Outcome:	The proposed model predicts four appraisal dimensions without significant loss . the proposed model is compared with enISEAR, a corpus of events created in an experimental setting and annotated for appraisal.

Copied to clipboard

Joosung Lee, Wooin Lee

Multimodal End-to-End Sparse Model for Emotion Recognition (2021.naacl-main)

Challenge:	Emotion recognition in conversation is inaccurate if the previous utterances are not taken into account, so many studies reflect the dialogue context to improve the performance.
Approach:	They propose a method that combines pre-trained memory with the context model to improve the performance of the context models.
Outcome:	The proposed method achieves the first or second performance on all data and is state-of-the-art among systems that do not leverage structured data.

Copied to clipboard

Wenliang Dai, Samuel Cahyawijaya, Zihan Liu, Pascale Fung

Deriving Entity-Specific Embeddings from Multi-Entity Sequences (2024.lrec-main)

Challenge:	Existing work in emotion recognition uses a two-phase pipeline, but the extracted features are fixed and cannot be fine-tuned on different tasks.
Approach:	They propose a two-phase pipeline for emotion recognition and personality recognition . they propose restructured datasets to enable fully end-to-end training .
Outcome:	The proposed model outperforms the current state-of-the-art models on emotion recognition and personality recognition tasks with half less computation in the feature extraction part.

Copied to clipboard

Connor Heaton, Prasenjit Mitra

BRIGHTER: BRIdging the Gap in Human-Annotated Textual Emotion Recognition Datasets for 28 Languages (2025.acl-long)

Challenge:	Existing methods toward entity-specific prediction involve redundant computation or post-processing outside of the transformer.
Approach:	They propose a method for deriving entity-specific embeddings from a multi-entity sequence completely within the transformer, with a loose definition of entity amenable to many problem spaces.
Outcome:	The proposed method can be applied to emotion recognition in conversation and player performance projection in baseball and achieve SOTA in both.

Copied to clipboard

Shamsuddeen Hassan Muhammad, Nedjma Ousidhoum, Idris Abdulmumin, Jan Philip Wahle, Terry Ruas, Meriem Beloucif, Christine de Kock, Nirmal Surange, Daniela Teodorescu, Ibrahim Said Ahmad, David Ifeoluwa Adelani, Alham Fikri Aji, Felermino D. M. A. Ali, Ilseyar Alimova, Vladimir Araujo, Nikolay Babakov, Naomi Baes, Ana-Maria Bucur, Andiswa Bukula, Guanqun Cao, Rodrigo Tufiño, Rendi Chevi, Chiamaka Ijeoma Chukwuneke, Alexandra Ciobotaru, Daryna Dementieva, Murja Sani Gadanya, Robert Geislinger, Bela Gipp, Oumaima Hourrane, Oana Ignat, Falalu Ibrahim Lawan, Rooweither Mabuya, Rahmad Mahendra, Vukosi Marivate, Alexander Panchenko, Andrew Piper, Charles Henrique Porto Ferreira, Vitaly Protasov, Samuel Rutunda, Manish Shrivastava, Aura Cristina Udrea, Lilian Diana Awuor Wanzare, Sophie Wu, Florian Valentin Wunderlich, Hanif Muhammad Zhafran, Tianhui Zhang, Yi Zhou, Saif M. Mohammad

EmoWOZ: A Large-Scale Corpus and Labelling Scheme for Emotion Recognition in Task-Oriented Dialogue Systems (2022.lrec-1)

Challenge:	Emotion recognition is an umbrella term for several NLP tasks, but most work on high-resource languages has focused on low-resourced languages.
Approach:	They propose to use emotion recognition to describe perceived emotions in 28 different languages and across several domains to identify and annotate the datasets.
Outcome:	The proposed datasets cover low-resource languages from Africa, Asia, Eastern Europe, and Latin America, with instances labeled by fluent speakers.

Copied to clipboard

Shutong Feng, Nurul Lubis, Christian Geishauser, Hsien-chin Lin, Michael Heck, Carel van Niekerk, Milica Gasic

Natural Language Interfaces

Challenge:	Existing emotion-annotated task-oriented corpora are limited in size, label richness, and public availability, creating a bottleneck for downstream tasks.
Approach:	They propose a large-scale manually emotion-annotated corpus of task-oriented dialogues based on a multi-domain task-orientated dataset.
Outcome:	The proposed method is based on a task-oriented dialogue dataset with 11K dialogues and 83K emotion annotations of user utterances.

EmotionTalk: An Interactive Chinese Multimodal Emotion Dataset With Rich Annotations (2026.findings-acl)

Copied to clipboard

Haoqin Sun, Jinghua Zhao, Xuechen Wang, Shiwan Zhao, Jiaming Zhou, Hui Wang, Xi Yang, Yequan Wang, Yonghua Lin

AffectCodec: Emotion-Preserving Neural Speech Codec for Expressive Speech Modeling (2026.findings-acl)

Challenge:	Existing datasets face issues such as low quality, limited scale, and incomplete modalities, hindering model performance.
Approach:	They propose to use Chinese multimodal datasets to capture authentic emotional interplay from 19 professional actors.
Outcome:	The EmotionTalk dataset spans 23.6 hours of dyadic conversations across diverse scenarios.

Copied to clipboard

Jiacheng Shi, Hongfei Du, Xinyuan Song, Y. Alicia Hong, Yanfu Zhang, Ashley Gao

Emotion Recognition in Conversation via Dynamic Personality (2024.lrec-main)

Challenge:	Existing codecs optimize acoustic reconstruction, leaving emotion expressiveness insufficiently modeled at the representation level.
Approach:	They propose an emotion-guided neural speech codec that preserves emotional information while maintaining semantic fidelity and prosodic naturalness.
Outcome:	The proposed codec preserves emotional cues while maintaining semantic fidelity and prosodic naturalness.

Copied to clipboard

Yan Wang, Bo Wang, Yachao Zhao, Dongming Zhao, Xiaojia Jin, Jijun Zhang, Ruifang He, Yuexian Hou

UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition (2022.emnlp-main)

Challenge:	Existing approaches to ERC focus on conversational contexts, but focus on static personality.
Approach:	They propose a model that considers the dynamic personality of speakers during conversations.
Outcome:	The proposed model outperforms existing models on three benchmark conversational datasets.

Copied to clipboard

Guimin Hu, Ting-En Lin, Yi Zhao, Guangming Lu, Yuchuan Wu, Yongbin Li

CoMuMDR: Code-mixed Multi-modal Multi-domain corpus for Discourse paRsing in conversations (2025.findings-acl)

Challenge:	Existing studies study sentiment and emotion separately and do not fully exploit the complementary knowledge behind the two.
Approach:	They propose a multimodal sentiment knowledge-sharing framework that unifies MSA and ERC tasks from features, labels, and models.
Outcome:	The proposed framework achieves consistent improvements on four public benchmark datasets on MOSI, MOSEI, MELD, and IEMOCAP.

Copied to clipboard

Divyaksh Shukla, Ritesh Baviskar, Dwijesh Gohil, Aniket Tiwari, Atul Shree, Ashutosh Modi

Ethics Sheets for AI Tasks (2022.acl-long)

Challenge:	Discourse parsing datasets based on conversations are restricted to a single domain . a lack of discourse structures in audio-based conversations is a challenge .
Approach:	They introduce CoMuMDR: Code-mixed Multi-modal Multi-domain corpus for Discourse parsing in conversations.
Outcome:	The proposed corpus is code-mixed in Hindi and English and annotated with nine discourse relations.

Copied to clipboard

Saif Mohammad

Responsible & Trustworthy NLP

Challenge:	a recent study has shown that technology can lead to more adverse outcomes for marginalized populations . a new effort is called Ethics Sheets for AI Tasks to flesh out ethical considerations .
Approach:	a new effort will focus on ethical considerations at the level of AI tasks . authors propose a template for ethics sheets with 50 ethical consideration examples .
Outcome:	a new form of ethics sheets for AI tasks aims to flesh out assumptions and ethical considerations hidden in how a task is commonly framed . a template for ethics sheets with 50 ethical consideration, using the task of emotion recognition as an example, will be presented .

Relation-aware Graph Attention Networks with Relational Position Encodings for Emotion Recognition in Conversations (2020.emnlp-main)

Copied to clipboard

Taichi Ishiwatari, Yuki Yasuda, Taro Miyazaki, Jun Goto

Supervised Adversarial Contrastive Learning for Emotion Recognition in Conversations (2023.acl-long)

Challenge:	Recent research on emotion recognition in conversations (ERC) does not take self-dependency or inter-speaker dependency into account.
Approach:	They propose a relational graph attention network (RGAT) model that takes speaker dependency and sequential information into account by encoding the relational Graph structure.
Outcome:	The proposed model outperforms the state-of-the-art on four ERC datasets.

Copied to clipboard

Dou Hu, Yinan Bao, Lingwei Wei, Wei Zhou, Songlin Hu

MuCDN: Mutual Conversational Detachment Network for Emotion Recognition in Multi-Party Conversations (2022.coling-1)

Challenge:	Existing methods to recognize emotions have limitations in discovering the intrinsic structure of data relevant to emotion labels, and struggle to extract generalized and robust representations.
Approach:	They propose a supervised adversarial contrastive learning framework for learning class-spread structured representations in a controlled manner.
Outcome:	The proposed framework can extract generalized and robust representations on three datasets and achieves state-of-the-art performance.

Copied to clipboard

Weixiang Zhao, Yanyan Zhao, Bing Qin

EmoInHindi: A Multi-label Emotion and Intensity Annotated Dataset in Hindi for Emotion Recognition in Dialogues (2022.lrec-1)

Challenge:	Emotion recognition in multi-party conversations is a challenging task that predicts the emotion for each utterance.
Approach:	They propose to separate conversations into detached threads to capture emotional clues in conversational context . they propose to use mutual detachment networks to perform context and speaker-specific modeling within detached thread.
Outcome:	The proposed model outperforms baseline models on two datasets.

Copied to clipboard

Gopendra Vikram Singh, Priyanshu Priya, Mauajama Firdaus, Asif Ekbal, Pushpak Bhattacharyya

Natural Language Interfaces

Challenge:	Existing datasets for emotion recognition in dialogues are in English . existing datasets are limited to a few languages like Hindi .
Approach:	They propose a large conversational dataset in Hindi for multi-label emotion and intensity recognition in conversations . they use a Wizard-of-Oz manner to annotate dialogues with 16 emotion labels .
Outcome:	The proposed dataset contains 1,814 dialogues with 44,247 utterances in Hindi . it is based on a Wizard-of-Oz manner and can detect emotions in conversation .

CLASP: Cross-modal Alignment Using Pre-trained Unimodal Models (2024.findings-acl)

Copied to clipboard

Jianing Zhou, Ziheng Zeng, Hongyu Gong, Suma Bhat

A Cross-Modality Context Fusion and Semantic Refinement Network for Emotion Recognition in Conversation (2023.acl-long)

Challenge:	Recent advances in speech-text pretraining rely on parallel speech- text data . however, data accessibility is a challenge due to the limited data available.
Approach:	They propose a framework for jointly performing speech and text processing without parallel corpora during pre-training but only downstream.
Outcome:	The proposed framework extracts distinct representations for speech and text, aligning them effectively in a newly defined space using a multi-level contrastive learning mechanism.

Copied to clipboard

Xiaoheng Zhang, Yang Li

QAP: A Quantum-Inspired Adaptive-Priority-Learning Model for Multimodal Emotion Recognition (2023.findings-acl)

Challenge:	Emotion recognition in conversation studies focus on textual modality, but they lack contextual information and focus on a limited number of modalities.
Approach:	They propose a cross-modal context fusion and semantic refinement network to explore multimodal interactions and a graph-based semantic refinements transformer to solve the limitation of insufficient semantic relationship information between utterances.
Outcome:	The proposed method is compared with other state-of-the-art methods on two public benchmark datasets and shows its potential for emotion recognition.

Copied to clipboard

Ziming Li, Yan Zhou, Yaxin Liu, Fuqing Zhu, Chuanpeng Yang, Songlin Hu

Task-Aware Self-Supervised Framework for Dialogue Discourse Parsing (2023.findings-emnlp)

Challenge:	Experimental results show that multimodal emotion recognition is a state-of-the-art technique . textual, visual and acoustic modalities are involved in multimodal video emotion recognition .
Approach:	They propose a quantum-inspired adaptive-priority-learning model to address the challenges . they use quantum state to model modal features and Q-attention to integrate three modalities .
Outcome:	Experimental results show that QAP improves on previous models.

Copied to clipboard

Wei Li, Luyao Zhu, Wei Shao, Zonglin Yang, Erik Cambria

Humans Hallucinate Too: Language Models Identify and Correct Subjective Annotation Errors With Label-in-a-Haystack Prompts (2025.emnlp-main)

Challenge:	Existing discourse parsing approaches are constrained by predefined relation types, which can impede the adaptability of the parser for downstream tasks.
Approach:	They propose to introduce a task-aware paradigm to improve the versatility of the parser.
Outcome:	Empirical studies on dialogue discourse parsing datasets and a downstream task demonstrate the proposed framework.

Copied to clipboard

Georgios Chochlakis, Peter Wu, Tikka Arjun Singh Bedi, Marcus Ma, Kristina Lerman, Shrikanth Narayanan

Joyful: Joint Modality Fusion and Graph Contrastive Learning for Multimoda Emotion Recognition (2023.emnlp-main)

Challenge:	Existing approaches to model complex subjective tasks in natural language are limited by significant variation in annotations.
Approach:	They propose a simple in-context learning binary filtering baseline that estimates the reasonableness of a document-label pair.
Outcome:	The proposed approach can be integrated into annotation pipelines to enhance signal-to-noise ratios.

Copied to clipboard

Dongyuan Li, Yusong Wang, Kotaro Funakoshi, Manabu Okumura

Federated Meta-Learning for Emotion and Sentiment Aware Multi-modal Complaint Identification (2023.emnlp-main)

Challenge:	Existing graph-based methods fail to depict global contextual features and local diverse unimodal features in a dialogue.
Approach:	They propose a method for joint modality fusion and graph contrastive learning for multimodal emotion recognition using a multimodal fusion mechanism and a graph contrastative learning framework.
Outcome:	The proposed method improves multimodal emotion recognition on unbalanced and small-scale emotional datasets.

Copied to clipboard

Apoorva Singh, Siddarth Chandrasekar, Sriparna Saha, Tanmay Sen

EMO-RL: Emotion-Rule-Based Reinforcement Learning Enhanced Audio-Language Model for Generalized Speech Emotion Recognition (2025.findings-emnlp)

Challenge:	Existing studies on complaint identification are limited to text.
Approach:	They propose a meta-learning-based multi-modal multi-task framework for identifying complaints using emotion recognition and sentiment analysis as auxiliary tasks.
Outcome:	The proposed framework outperforms baselines and state-of-the-art approaches in centralized and federated meta-learning settings.

Copied to clipboard

Pengcheng Li, Botao Zhao, Zuheng Kang, Junqing Peng, Xiaoyang Qu, Yayun He, Jianzong Wang

Generative Error Correction for Emotion-aware Speech-to-text Translation (2025.findings-acl)

Challenge:	Recent advances in reinforcement learning (RL) have shown promise in improving LALMs’ reasoning abilities, but their performance in affective computing tasks remains suboptimal.
Approach:	They propose a framework incorporating reinforcement learning with two key innovations: Emotion Similarity-Weighted Reward (ESWR) and Explicit Structured Reasoning (ESR).
Outcome:	The proposed framework improves LALMs' reasoning abilities on MELD and IEMOCAP datasets and shows strong generalization.

Copied to clipboard

Zhengdong Yang, Sheng Li, Chenhui Chu

Text Generation

Challenge:	Despite recent advances in speech-to-text translation, the impact of the emotion content has been overlooked.
Approach:	They propose to use generative error correction (GER) to generate the translation based on the decoded N-best hypotheses and combine emotion and sentiment labels into the LLM finetuning process to enable the model to consider the emotion content.
Outcome:	The proposed model can translate speech in English-Chinese using GER and emotion and sentiment labels.

MERMAID: Multi-perspective Self-reflective Agents with Generative Augmentation for Emotion Recognition (2025.emnlp-main)

Copied to clipboard

Zhongyu Yang, Junhao Song, Siyang Song, Wei Pang, Yingfang Yuan

Do Emotions Influence Moral Judgment in Large Language Models? (2026.findings-acl)

Challenge:	Existing multimodal large language models struggle to handle ambiguous emotional expressions and implicit affective cues, which are crucial for affective understanding but largely overlooked.
Approach:	They propose a multi-agent framework that integrates a self-reflection module, an emotion-guided visual augmentation module, and a cross-modal verification module to enhance emotion recognition.
Outcome:	Extensive experiments show that MERMAID outperforms existing methods and achieves absolute accuracy gains of 8.70%–27.90% across diverse benchmarks.

Copied to clipboard

Mohammad Saim, Tianyu Jiang